Method and system for unsynchronized structured lighting

ABSTRACT

A system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting is disclosed. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate, and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene. Each of the synchronized image frames corresponds to one image pattern of the sequence of image patterns.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to earlier filed U.S. ProvisionalApplication Ser. No. 61/949,529, filed Mar. 7, 2014, the contents ofwhich are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This patent document relates generally to the field of three-dimensionalshape capture of the surface geometry of an object, and moreparticularly to structured lighting three-dimensional shape capture.

2. Background of the Related Art

Three-dimensional scanning and digitization of the surface geometry ofobjects is commonly used in many industries and services, and theirapplications are numerous. A few examples of such applications areinspection and measurement of shape conformity in industrial productionsystems, digitization of clay models for industrial design and stylingapplications, reverse engineering of existing parts with complexgeometry for three-dimensional printing, interactive visualization ofobjects in multimedia applications, three-dimensional documentation ofartwork, historic and archaeological artifacts, human body scanning forbetter orthotics adaptation, biometry or custom-fit clothing, andthree-dimensional forensic reconstruction of crime scenes.

One technology for three-dimensional shape capture is based onstructured lighting. Three dimensional shape capture systems based onstructure lighting are more accurate than those based on time-of-flight(TOF) image sensors. In a standard structured lighting 3D shape capturesystem a pattern projector is used to illuminate the scene of interestwith a sequence of known two-dimensional patterns, and a camera is usedto capture a sequence of images, synchronized with the projectedpatterns. The camera captures one image for each projected pattern. Eachsequence of images captured by the camera is decoded by a computerprocessor into a dense set of projector-camera pixel correspondences,and subsequently into a three-dimensional range image, using theprinciples of optical triangulation.

The main limitation of three-dimensional shape capture systems is therequired synchronization between projector and camera. To capture athree-dimensional snapshot of a moving scene, the sequence of patternsmust be projected at a fast rate, the camera must capture image framesexactly at the same frame rate, and the camera has to start capturingthe first frame of the sequence exactly when the projector starts toproject the first pattern.

Therefore, there is a need for three-dimensional shape measurementmethods and systems based on structure lighting where the camera and thepattern projector are not synchronized.

Further complicating matters, image sensors generally use one of twodifferent technologies to capture an image, referred to as “rollingshutter” and “global shutter”. “Rolling shutter” is a method of imagecapture in which a still picture or each frame of a video is capturednot by taking a snapshot of the entire scene at single instant in timebut rather by scanning across the scene rapidly, either vertically orhorizontally. In other words, not all parts of the image of the sceneare recorded at exactly the same instant. This is in contrast with“global shutter” in which the entire frame is captured at the sameinstant. Even though most image sensors in consumer devices are rollingshutter sensors, many image sensors used in industrial applications areglobal shutter sensors.

Therefore, there is a need for three-dimensional shape measurementmethods and systems based on structure lighting where the camera and thepattern projector are not synchronized, supporting both global shutterand rolling shutter image sensors.

SUMMARY OF THE INVENTION

A system and method to capture the surface geometry a three-dimensionalobject in a scene using unsynchronized structured lighting solves theproblems of the prior art. The method and system includes a patternprojector configured and arranged to project a sequence of imagepatterns onto the scene at a pattern frame rate, a camera configured andarranged to capture a sequence of unsynchronized image patterns of thescene at an image capture rate; and a processor configured and arrangedto synthesize a sequence of synchronized image frames from theunsynchronized image patterns of the scene, each of the synchronizedimage frames corresponding to one image pattern of the sequence of imagepatterns. Because the method enables use of an unsynchronized patternprojector and camera significant cost savings can be achieved. Themethod enables use of inexpensive cameras, such as smartphone cameras,webcams, point-and-shoot digital cameras, camcorders as well asindustrial cameras. Furthermore, the method and system enable processingthe images with a variety of computing hardware, such as computers,digital signal processors, smartphone processors and the like.Consequently, three-dimensional image capture using structured lightingmay be used with relatively little capital investment.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the method andsystem will become better understood with reference to the followingdescription, appended claims, and accompanying drawings where:

FIG. 1 is an illustration of an exemplary embodiment of the method andsystem for unsynchronized structured lighting;

FIG. 2 shows a flowchart of an exemplary embodiment of the method andsystem for unsynchronized structured lighting;

FIG. 3 shows a flowchart of an exemplary embodiment of the method andsystem for unsynchronized structured lighting of synthesizing thesynchronized sequence of image frames;

FIG. 4 is a chart illustrating the timing for a method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, where a global shutter image sensor is used and where theimage frame rate is identical to the pattern frame rate;

FIG. 5 is a chart illustrating the timing for a method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, where a rolling shutter image sensor is used and where theimage frame rate is identical to the pattern frame rate;

FIG. 6 is a chart illustrating the timing for a method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, where a global shutter image sensor is used and where theimage frame rate is higher or equal than the pattern frame rate

FIG. 7 is a chart illustrating the timing for method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, where a rolling shutter image sensor is used and where theimage frame rate is higher or equal than the pattern frame rate;

FIG. 8 shows a chart illustrating a calibration pattern used to correctthe time;

FIG. 9 shows a chart illustrating image data normalized according to themethod described herein; and

FIG. 10 shows a chart illustrating a model estimating the pattern valuefor each pixel in a captured image.

DESCRIPTION OF THE PREFERRED EMBODIMENT

A system and method to capture the surface geometry a three-dimensionalobject in a scene using unsynchronized structured lighting is showngenerally in FIGS. 1 and 2. The method and system includes a patternprojector configured and arranged to project a sequence of imagepatterns onto the scene at a pattern frame rate, a camera configured andarranged to capture a sequence of unsynchronized image patterns of thescene at an image capture rate; and a processor configured and arrangedto synthesize a sequence of synchronized image frames from theunsynchronized image patterns of the scene, each of the synchronizedimage frames corresponding to one image pattern of the sequence of imagepatterns.

One object of the present invention is a system to synthesize asynchronized sequence of image frames from an unsynchronized sequence ofimage frames, illustrated in FIG. 1; the unsynchronized image framescaptured while a three-dimensional scene was illuminated by a sequenceof patterns. The system comprises a pattern projector 2020 and a camera2030. An object 2010 is partially illuminated by the pattern projector2020, and partially visible by the camera 2030. The pattern projectorprojects a sequence of patterns 2021, 2022, 2023 at a certain patternrate. The pattern rate is measured in patterns per second. The camera2030 captures a sequence of unsynchronized image frames 2031, 2032,2033, 2034, at a certain frame rate. The frame rate is larger or equalthan the pattern rate. The number of unsynchronized image framescaptured by the camera is larger or equal than the number of patterns inthe sequence of patterns. The camera starts capturing the firstunsynchronized image frame not earlier than the time when the projectorstarts projecting the first pattern of the sequence of image patterns.The camera ends capturing the last unsynchronized image frame notearlier than the time when the projector ends projecting the lastpattern of the sequence of image patterns. To capture a frame the cameraopens the camera aperture, which it closes after an exposure time. Thecamera determines the pixel values by integrating the incoming lightwhile the aperture is open. Since camera and projector are notsynchronized, the projector may switch patterns while the camera has theaperture open. As a result, some or all of the pixels of the capturedunsynchronized image frame 2032 will be partially exposed to twoconsecutive patterns 2021 and 2022. The resulting sequence ofunsynchronized image frames, are transmitted to a computer processor2040, which executes a method to synthesize a synchronized sequence ofimage frames from the unsynchronized sequence of image frames. Thenumber of frames in the synchronized sequence of image frames will bethe same as the number of patterns and represent estimates of what thecamera would have captured if it were synchronized with the projector.In a preferred embodiment the camera 2030 and the computer processor2040 are components of a single device such a digital camera,smartphone, or computer tablet.

Another object of the invention is an unsynchronized three-dimensionalshape capture system, comprising the system to synthesize a synchronizedsequence of image frames from an unsynchronized sequence of image framesdescribed above, and further comprising prior art methods for decoding,three-dimensional triangulation, and optionally geometric processing,executed by the computer processor.

Another object of the invention is a three-dimensional snapshot cameracomprising the unsynchronized three-dimensional shape capture system,where the projector has the means to select the pattern rate from aplurality of supported pattern rates, the camera has the means to selectthe frame rate from a plurality of supported frame rates, and the camerais capable of capturing the unsynchronized image frames in burst mode ata fast frame rate. In a preferred embodiment the projector has a knob toselect the pattern rate. In another preferred embodiment the patternrate is set by a pattern rate code sent to the projector through acommunications link. Furthermore, the system has means to set thepattern rate and the frame rate so that the frame rate is not slowerthan the pattern rate. In a more preferred embodiment the user sets thepattern rate and the frame rate.

In a more preferred embodiment of the snapshot camera, the camera hasthe means to receive a camera trigger signal, and the means to set thenumber of burst mode frames. In an even more preferred embodiment, thecamera trigger signal is generated by a camera trigger push-button. Whenthe camera receives the trigger signal it starts capturing theunsynchronized image frames at the set frame rate, and it stopscapturing unsynchronized image frames after capturing the set number ofburst mode frames.

In a first preferred embodiment of the snapshot camera with cameratrigger signal, the projector continuously projects the sequence ofpatterns in a cyclic fashion. In a more preferred embodiment the systemhas the means of detecting when the first pattern is about to beprojected, and the camera trigger signal is delayed until that moment.

In a second preferred embodiment of the snapshot camera with cameratrigger signal, the projector has the means to receive a projectortrigger signal. In a more preferred embodiment the camera generates theprojector trigger signal after receiving the camera trigger signal, andthe camera has the means to send the projector trigger signal to theprojector. In an even more preferred embodiment the camera has a flashtrigger output, and it sends the projector trigger signal to theprojector through the flash trigger output. When the projector receivesthe trigger signal it starts projecting the sequence of patterns at theset pattern rate, and it stops projecting patterns after it projects thelast pattern.

Another object of this invention is a method to synthesize asynchronized sequence of image frames from an unsynchronized sequence ofimage frames, generating a number of frames in the synchronized sequenceof image frames equal to the number of projected patterns, andrepresenting estimates of what the camera would have captured if it weresynchronized with the projector.

As will be described in greater detail below in the associated proofs,the

method to synthesize the synchronized sequence of image frames from theunsynchronized sequence of image frames is shown generally in FIG. 3.Each of the synchronized image frames corresponds to one of a sequenceof image patterns. Further, the synchronized image frames andunsynchronized image frames have the same width and height. Each imageframe includes a plurality of common pixels. In a first step a) theplurality of common pixels is partitioned into a pixel partition. Thepixel partition includes a plurality of pixel sets, each of which isdisjoint. Further each of the plurality of common pixels is a member ofone of the pixel sets. In a step b) a pixel set is selected. In a stepc) a pixel set measurement matrix is built. In a step d) a pixel setprojection matrix is estimated. The pixel set projection matrix projectsa measurement space vector onto the column space of the pixel setmeasurement matrix. In a step e) a pixel set system matrix is estimated.The system matrix is parameterized by a set of system matrix parameters.In a step f) a pixel set synchronized matrix as a function of the pixelset measurement matrix and the pixel set system matrix is estimated. Ina step g) steps b) to f) are repeated until all the pixels sets havebeen selected. In a step h) a sequence of synchronized image frames fromthe pixel set synchronized matrices is constructed.

In a preferred embodiment, the method to synthesize the synchronizedsequence of image frames from an unsynchronized sequence of imageframes, applies to a global shutter image sensor where the image framerate is identical to the pattern frame rate. FIG. 4 illustrates thetiming for this embodiment. In this embodiment, the projector projects Npatterns at a fixed frame rate, a global shutter image sensor capture Nimages at identical frame rate. Capturing each image takes exactly oneunit of time, normalized by the projector frame rate. The start time forthe first image capture t₀ is unknown, but the start time for the n-thimage capture is related to the start time for the first image capturet_(n)=t₀+n−1. The actual value measured by the image sensor at the (x,y) pixel of the n-th image, can be modeled as

I _(n)(x, y)=(1−t ₀)P _(n)(x, y)+t ₀ P _(n+1)(x, y)

where P_(n)(x, y) and P_(n−1)(x, y) represent the pattern values to beestimated that contribute to the image pixel (x, y) and P_(N+1)≡P₁.Projected patterns are known in advance, but since it is not known whichprojector pixel illuminates each image pixel, they have to be treated asunknown. To estimate the value of t₀, the following expression isminimized

${E( t_{0} )} = {\frac{1}{2}{\sum_{n = 1}^{N}{\sum\limits_{({x,y})}\; ( {{( {1 - t_{0}} ){P_{n}( {x,y} )}} + {t_{0}{P_{n + 1}( {x,y} )}} - {I_{n}( {x,y} )}} )^{2}}}}$

with respect to t₀ , where the sum is over a subset of pixels (x, y) forwhich the corresponding pattern pixel values P_(n)(x, y) and P_(n−1)(x,y) are known. Differentiating E(t₀) with respect to t₀, and equating theresult to zero, an expression to estimate t₀ is obtained

$t_{0} = \frac{\sum_{n = 1}^{N}{\sum_{({x,y})}{( {{P_{n + 1}( {x,y} )} - {P_{n}( {x,y} )}} )( {{I_{n}( {x,y} )} - {P_{n}( {x,y} )}} )}}}{\sum_{n = 1}^{N}{\sum_{({x,y})}( {{P_{n + 1}( {x,y} )} - {P_{n}( {x,y} )}} )^{2}}}$

Once the value of t₀ has been estimated, the N pattern pixel valuesP₁(x, y), . . . , P_(N)(x, y) can be estimated for each pixel (x, y) byminimizing the following expression

E(P₁(x, y), . . . , P_(N)(x, y))= ¹ ₂Σ_(n=1) ^(N)((1−t ₀)P _(n)(x, y)+t₀ P _(n+1)(x, y)−I _(n)(x, y))²

which reduces to solving the following system of N linear equations

βP _(n−1)(x, y)+αP _(n)(x, y)+βP _(n+1)(x, y)=t₀ I _(n−1)(x, y)+(1−t ₀)I_(n)(x, y)

for n=1 , . . ., N, where α=t² ₀+(1−t₀)² and ⊕=t₀(1−−t₀).

In another preferred embodiment, the method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, applies to a rolling shutter image sensor where the imageframe rate is identical to the pattern frame rate. FIG. 5 illustratesthe timing for this embodiment. We project N patterns at fixedframerate, a rolling shutter camera captures N images. Capture beginswhile pattern P₁ is being projected. Projector framerate is 1, patternP_(n) is projected between time n−1 and n. A camera frame is read everyt_(f) time, camera framerate is assumed equal to projector framerate butin practice may vary a little. A camera row requires t_(r) time to bereadout from the sensor, thus, a sensor with Y rows needs a time Yt_(r)to read a complete frame, Yt_(f)≦t_(r). Each camera frame is exposedt_(e) time, its readout begins immediately after exposure ends,t_(e)+t_(r)≦t_(f).

Camera row y in image n begins being exposed at time t_(n, y)

t _(n, y) =t ₀+(n−1)t_(f) +y t _(r) , y:0 . . . Y−1,

and exposition ends at time t_(n, y)+t_(e)

In this model image n is exposed while pattern P_(n) and P_(n+1) arebeing projected. Intensity level measured at a pixel in row y is givenby

I _(n, y)=(n−t _(n, y))k _(n, y) P _(n)+(t _(n, y) +t _(e) −n)k _(n, y)P _(n+1) +C _(n, y,)

The constants k_(n, y) and C_(n, y) are scene dependent.

Let be min {I_(n, y)} a pixel being being exposed while P(t)=0, and max{I_(n, y)} a pixel being exposed while P(t)=1, max{I_(n, y)}=t_(e)k_(n, y)+C_(n, y). Now, we define a normalized imageĨ_(n, y) as,

${\overset{\sim}{I}}_{n,y} = \frac{{( {n - t_{n,y}} )P_{n}} + {( {t_{n,y} + t_{e} - n} )P_{n + 1}}}{t_{e}}$

A normalized image is completely defined by the time variables andpattern values. In this section we want to estimate the time variables.Let's rewrite Equation 58 as

${\overset{\sim}{I}}_{n,y} = {{{- n}\; \Delta \; P_{n}\frac{1}{t_{e}}} + {( {n - 1} )\Delta \; P_{n}\frac{t_{f}}{t_{e}}} + {y\; \Delta \; P_{n}\frac{t_{r}}{t_{e}}} + {\Delta \; P_{n}\frac{t_{0}}{t_{e}}} + P_{n + 1}}$

being t₀ and d unknown. Image pixel values are given by

I _(n)(x, y)=(1−t ₀ −yd)P _(n)(x, y)+(t ₀ +yd)P _(n)+1(x, y).

Same as before, P_(n)(x, y) and P_(n+1) (x, y) represent the patternvalues contributing to camera pixel (x, y), we define P_(N+)1≡P₁,P₀≡P_(N), I_(N+1)≡I₁, and I₀≡I_(N), and I will omit pixel (x, y) tosimplify the notation. We now minimize the following energy to find thetime variables t₀ and d

${E( {t_{0},d} )} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; ( {{( {1 - t_{0} - {yd}} )P_{n}} + {( {t_{0} + {yd}} )P_{n + 1}} - I_{n}} )^{2}}}}$

The partial derivatives are given by

$\frac{\partial{E( {t_{0},d} )}}{\partial t_{0}} = {\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; {( {P_{n + 1} - P_{n}} )( {{( {1 - t_{0} - {yd}} )P_{n}} + {( {t_{0} + {yd}} )P_{n + 1}} - I_{n}} )}}}$$\frac{\partial{E( {t_{0},d} )}}{\partial d} = {\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; {{y( {P_{n + 1} - P_{n}} )}( {{( {1 - t_{0} - {yd}} )P_{n}} + {( {t_{0} + {yd}} )P_{n + 1}} - I_{n}} )}}}$

We set the gradient equal to the null vector and reorder as

$\begin{bmatrix}t_{0} \\d\end{bmatrix} = {( {\sum\limits_{n,x,y}\; {( {P_{n + 1} - P_{n}} )^{2}\begin{bmatrix}1 & y \\y & y^{2}\end{bmatrix}}} )^{- 1}( {\sum\limits_{n,x,y}\; {( {P_{n + 1} - P_{n}} ){( {I_{n} - {Pn}} )\begin{bmatrix}1 \\y\end{bmatrix}}}} )}$

We use Equation 29 to compute t₀ and d when we have some known (orestimated) pattern values.

With known t₀ and d we estimate pattern values minimizing

${E( {P_{1},\ldots \mspace{14mu},P_{N}} )} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; {( {{( {1 - t_{0} - {yd}} )P_{n}} + {( {t_{0} + {yd}} )P_{n + 1}} - I_{n}} )^{2}.}}}$

Analogous as in Case 1 we obtain that Ap=b with A as in Equation 12 andα, β, and b defined as

α=(1t ₀ −yd)²+(t ₀ +yd)², β=(1−t ₀ −yd)(t ₀ +yd)

b=(1−t ₀ −yd)(I₁ I ₂ . . . , I _(N))^(T)+(t ₀ +yd)(I _(N) , I ₁ , . . .I _(N−1))^(T)

Pattern values for each pixel are given by p=A⁻¹ b.

In another preferred embodiment, the method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, applies to a global shutter image sensor where the imageframe rate is higher or equal than the pattern frame rate. FIG. 6illustrates the timing for this embodiment. We now project M patterns atfixed framerate and we capture N images with a global shutter camera,also at a fixed framerate. We require that N≧M. Capture begins whilepattern P₁ is being projected. We introduce a new variable d which isthe camera capture delay from one row to the next. Same as in Case 1, upto two patterns may contribute to each image but here we do not knowwhich ones are because the camera framerate is unknown. The new imageequation is

$I_{n} = {{\int_{t_{n - 1}}^{t_{n}}{\sum\limits_{m = 1}^{M}\; {{f_{m}(t)}P_{m}\ {t}}}} = {\sum\limits_{m = 1}^{M}\; {P_{m}{\int_{t_{n - 1}}^{t_{n}}{{f_{m}(t)}\ {t}}}}}}$${f_{m}(t)} = \{ \begin{matrix}1 & {{{{if}\mspace{14mu} m} - 1} \leq t \leq m} \\0 & {otherwise}\end{matrix} $

Let be Δt≡t_(n−1)−t_(n) the time between image frames, let be p=(P₁, . .. , P_(M))^(T) and Φ_(n)(t₀, Δt)=(Φ(n, 1, t₀, Δt), . . . , Φ(n, M, t₀,Δt))^(T) and rewrite Equation 33 as

$I_{n} = {\frac{1}{\Delta \; t}{\varphi_{n}( {t_{0},{\Delta \; t}} )}^{T}p}$

Each function Φ(n, m, t0, Δt)=∫_(t) _(n−1) ^(t) ^(n) f_(m)(t)dt can bewritten as

Φ(n, m, t0, Δt)=max(0, min(m, t _(n))−max(m−1, t _(n−1)))

Same as before, P_(n)(x, y) represents a pattern value contributing tocamera pixel (x, y), we define P_(N+)1≡P₁, P₀ ≡P_(N), I_(N+1)≡I₁, andI₀≡I_(N), and I will omit pixel (x, y) to simplify the notation.

We now minimize the following energy to find the time variables t₀ andΔt

${E( {t_{0},{\Delta \; t}} )} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; ( {{\frac{1}{\Delta \; t}{\varphi_{n}( {t_{0},{\Delta \; t}} )}^{T}p} - I_{n}} )^{2}}}}$

We solve for t₀ and Δt by making ∇E(t₀, Δt)=0

${\bigtriangledown \; {E( {t_{0},{\Delta \; t}} )}} = {\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; {\frac{1}{\Delta \; t}J\; {\varphi_{n}( {t_{0},{\Delta \; t}} )}^{T}{p( {{\frac{1}{\Delta \; t}{\varphi_{n}( {t_{0},{\Delta \; t}} )}^{T}p} - I_{n}} )}}}}$

Because JΦ_(n)(t₀, Δt) depends on the unknown value t=(t₀, Δt)^(T) wesolve for them iteratively

t^((i + 1)) = A_(t)(t^((i)))⁻¹b_(t)(t^((i)))${A_{t}(t)} = {\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; {\frac{1}{\Delta \; t^{2}}J\; {\varphi_{n}(t)}^{T}{pp}^{T}{V_{A}( {n,t} )}}}}$${b_{t}(t)} = {\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; {\frac{1}{\Delta \; t}J\; {\varphi_{n}(t)}^{T}{p( {I_{n} - {\frac{1}{\Delta \; t}{V_{b}( {n,t} )}^{T}p}} )}}}}$

Matrix V_(A)(n, t) and vector V_(b)(n, t) are defined such as

${\varphi_{n}(t)} = {{{V_{A}( {n,t} )}\begin{bmatrix}t_{0} \\{\Delta \; t}\end{bmatrix}} + {V_{b}( {n,t} )}}$

For completeness we include the following definitions:

  V_(A)(n, t) = [v_(A)(n, 1, t),  …  , v_(A)(n, M, t)]^(T)  V_(b)(n, t) = [v_(b)(n, 1, t),  …  , v_(b)(n, M, t)]^(T)  t_(diff) ≡ min (m, t_(n)) − max (m − 1, t_(n − 1))${v_{A}( {n,m,t} )} = \{ {{\begin{matrix}\lbrack {0,1} \rbrack^{T} & {{{{if}\mspace{14mu} m} - 1} \leq t_{n - 1} \leq t_{n} \leq {m\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\\lbrack {{- 1},{1 - n}} \rbrack^{T} & {{{{if}\mspace{14mu} m} - 1} \leq t_{n - 1} \leq m \leq {t_{n}\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\\lbrack {1,n} \rbrack^{T} & {{{if}\mspace{14mu} t_{n - 1}} \leq {m - 1} \leq t_{n} \leq {m\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\\lbrack {0,0} \rbrack^{T} & {{{if}\mspace{14mu} t_{n - 1}} \leq {m - 1} \leq m \leq {t_{n}\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\\lbrack {0,0} \rbrack^{T} & {otherwise}\end{matrix}{v_{b}( {n,m,t} )}} = \{ {{\begin{matrix}0 & {{{{if}\mspace{14mu} m} - 1} \leq t_{n - 1} \leq t_{n} \leq {m\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\m & {{{{if}\mspace{14mu} m} - 1} \leq t_{n - 1} \leq m \leq {t_{n}\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\{1 - m} & {{{if}\mspace{14mu} t_{n - 1}} \leq {m - 1} \leq t_{n} \leq {m\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\1 & {{{if}\mspace{14mu} t_{n - 1}} \leq {m - 1} \leq m \leq {t_{n}\mspace{14mu} {and}\mspace{14mu} t_{diff}} > 0} \\0 & {otherwise}\end{matrix}\mspace{20mu} J\; {\varphi_{n}(t)}} = {V_{A}( {n,t} )}} } $

With known t₀ and 66 t we estimate pattern values minimizing

${E(p)} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; ( {{{\varphi_{n}(t)}^{T}p} - I_{n}} )^{2}}}$

Analogous as in Case 1 we obtain that Ap=b with

A = Φ^(T)Φ, b = Φ^(T)I $\Phi = \begin{bmatrix}{\varphi_{1}(t)}^{T} \\\vdots \\{\varphi_{N}(t)}^{T}\end{bmatrix}$

Pattern values for each pixel are given by p=A⁻¹b.

In another preferred embodiment, the method to synthesize thesynchronized sequence of image frames from an unsynchronized sequence ofimage frames, applies to a rolling shutter image sensor where the imageframe rate is higher or equal than the pattern frame rate. FIG. 7illustrates the timing for this embodiment. Projector framerate is 1,pattern P_(m) is projected between time m−1 and m. A camera frame isread every t_(f) time. A camera row requires t_(r) time to be readoutfrom the sensor, thus, a sensor with Y rows needs a time Yt_(r) to reada complete frame, t_(f)≧Yt_(r). Each camera frame is exposed t_(e) time,its readout begins immediately after exposure ends, t_(e)+t_(r)≦t_(f).

Camera row y in image n begins being exposed at time t_(n, y)

t _(n, y) =t ₀+(n−1)t _(f) +y t _(r) , y:0 . . . Y−1

and exposition ends at time t_(n, y)+t_(e).

In this model a pixel intensity in image n at row y is given by

I_(n, y) = ∫_(t_(n, y))^(t_(n, y) + t_(e))k_(n, y)P(t) t + C_(n, y)$I_{n,y} = {{\sum\limits_{m = 1}^{M}\; {\max ( {0,{\int_{\max {({t_{n,y},{m - 1}})}}^{\min {({{t_{n,y} + t_{e}},m})}}{k_{n,y}P_{m}\ {t}}}} )}} + C_{n,y}}$

The constants k_(n, y) and C_(n, y) are scene dependent, P_(m) is either0 or 1.

Let be min{I_(n, y)} a pixel being exposed while P(t)=0, andmax{I_(n, y)} a pixel being exposed while P(t)=1,

min{I_(n, y)}=C_(n, y)

max{I _(n, y) }=t _(e) k _(n, y) +C _(n, y)

Now, we define a normalized image Ĩ_(n, y)as,

${\overset{\sim}{I}}_{n,y} = \frac{I_{n,y} - {\min \{ I_{n,y} \}}}{{\max \{ I_{n,y} \}} - {\min \{ I_{n,y} \}}}$${\overset{\sim}{I}}_{n,y} = {\frac{1}{t_{e}}{\sum\limits_{m = 1}^{M}\; {\max ( {0,{\int_{\max {({t_{n,y},{m - 1}})}}^{\min {({{t_{n,y} + t_{e}},m})}}{P_{m}\ {t}}}} )}}}$

A normalized image is completely defined by the time variables andpattern values. In this section we want to estimate the time variables.Let's rewrite the previous equation as,

$\mspace{20mu} {{\overset{\sim}{I}}_{n,y} = {\sum\limits_{m = 1}^{M}\; {{\varphi ( {n,m,y} )}P_{m}}}}$$\mspace{20mu} {{\varphi ( {n,m,y} )} = {\frac{1}{t_{e}}{\max ( {0,{{\min ( {{t_{n,y} + t_{e}},m} )} - {\max ( {t_{n,y},{m - 1}} )}}} )}}}$${{{{Let}\mspace{14mu} {be}\mspace{14mu} h} = {\frac{1}{t_{e}}( {1,t_{f},t_{r},t_{0}} )^{T}}},{{{now}\mspace{14mu} {we}\mspace{14mu} {write}\mspace{14mu} {\varphi ( {n,m,y} )}} = {v_{nmy}^{T}h\mspace{14mu} {with}\mspace{14mu} v_{nmy}\mspace{14mu} {defined}\mspace{14mu} {as}}}}\mspace{14mu}$$v_{nmy}^{T} = \{ {{{\begin{matrix}\lbrack {t_{e},0,0,0} \rbrack & {{{{if}\mspace{14mu} t_{n\; 2}} < t_{m\; 2}}{t_{n\; 1} > t_{m\; 1}}{t_{n\; 2} > t_{n\; 1}}} \\\lbrack {{t_{e} - m + 1},{n - 1},y,1} \rbrack & {{{{if}\mspace{14mu} t_{n\; 2}} < t_{m\; 2}}{t_{n\; 1} \leq t_{m\; 1}}{t_{n\; 2} > t_{m\; 1}}} \\\lbrack {m,{1 - n},{- y},{- 1}} \rbrack & {{{{if}\mspace{14mu} t_{n\; 2}} \geq t_{m\; 2}}{t_{n\; 1} > t_{m\; 1}}{t_{m\; 2} > t_{n\; 1}}} \\\lbrack {1,0,0,0} \rbrack & {{{{if}\mspace{14mu} t_{n\; 2}} \geq t_{m\; 2}}{t_{n\; 1} \leq t_{m\; 1}}{t_{m\; 2} > t_{m\; 1}}} \\\lbrack {0,0,0,0} \rbrack & {otherwise}\end{matrix}\mspace{20mu} t_{n\; 1}} = t_{n,y}},{t_{n\; 2} = {t_{n,y} + t_{e}}},{t_{m\; 1} = {m - 1}},{t_{m\; 2} = {{m\mspace{20mu} {Let}\mspace{14mu} {be}\mspace{14mu} p} = {{( {P_{1},\mspace{14mu} \ldots \mspace{14mu},P_{M}} )^{T}\mspace{14mu} {and}\mspace{14mu} {let}\mspace{14mu} V_{ny}\mspace{14mu} {be}\mspace{14mu} \mspace{20mu} V_{ny}} = {\begin{bmatrix}– & v_{n\; 1y}^{T} & – \\\; & \vdots & \; \\– & v_{nMy}^{T} & –\end{bmatrix}.}}}}} $

We now minimize the following energy to find the unknown h

${E(h)} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; {\sum\limits_{x,y}\; ( {{{p( {x,y} )}^{T}V_{ny}h} - {{\overset{\sim}{I}}_{n}( {x,y} )}} )^{2}}}}$

with the following constraints

$h > {\begin{bmatrix}0 \\1 \\0 \\0\end{bmatrix}\mspace{14mu} {{and}\mspace{14mu}\begin{bmatrix}0 & {- 1} & Y & 0 \\0 & {- 1} & 1 & 0\end{bmatrix}}\mspace{14mu} h} \leq \begin{bmatrix}0 \\{- 1}\end{bmatrix}$

or equivalently

$ {{\frac{t_{r}}{e} - \frac{t_{f}}{t_{e}}} \leq {- 1}}\Rightarrow{{t_{r} + t_{e}} \leq t_{f}} $

Equation E(h) cannot be minimized in closed form because the valuesmatrix V_(n, y) depends on the unknown values. Using an iterativeapproach the current value h^((i)) is used to compute V_(ny) ^((i)) andthe next value h^((i+1)pl .)

Up to this point we have assumed that the only unknown is h, meaningthat pattern values are known for all image pixels. The difficulty liesis knowing which pattern pixel is being observed by each camera pixel.We simplify this issue by making calibration patterns all ‘black or all‘white’, best seen in FIG. 8. For example, a sequence of four patterns‘{black, black, white, white}’ will produce images with completely blackand completely white pixels, as well as pixels in transition from blackto white and vice versa. The all black or white pixels are required toproduce normalized images, as shown in FIG. 9, and the pixels intransition constrain the solution of the parameter h in Equation E(h).

Decoding is done in two steps: 1) the time offset t₀ need to beestimated for this particular sequence; 2) the pattern values areestimated for each camera pixel, as shown in FIG. 10. Value t₀ isestimated using Equation E(h) where the known components of h are fixed,but some pattern values are required to be known, specially we need toknow for some pixels whether they are transitioning from ‘black’ to‘white’ or the opposite. Non-transitioning pixels provided noinformation in this step. Until now, we have projected a couple ofblack's and white's at the beginning of the sequence to ensure we cannormalized all pixels correctly and to simplify t₀ estimation. We willrevisit this point in the future for other pattern sequences.

Similarly as for the time variables, pattern values are estimated by

minimizing the following energy

${{E(p)} = {\frac{1}{2}{\sum\limits_{n = 1}^{N}\; ( {{h^{T}V_{ny}^{T}{p( {x,y} )}} - {{\overset{\sim}{I}}_{n}( {x,y} )}} )^{2}}}},{{s.t.\mspace{14mu} {p_{m}}} \leq 1}$

The matrix h^(T)V_(ny) ^(T) is bi-diagonal for N=M and it is fixed if his known.

Therefore, it can be seen that the exemplary embodiments of the methodand system provides a unique solution to the problem of using structurelighting for three-dimensional image capture where the camera andprojector are unsychronized.

It would be appreciated by those skilled in the art that various changesand modifications can be made to the illustrated embodiments withoutdeparting from the spirit of the present invention. All suchmodifications and changes are intended to be within the scope of thepresent invention except as limited by the scope of the appended claims.

What is claimed is:

1. A system to capture the surface geometry a three-dimensional objectin a scene, comprising: a pattern projector configured and arranged toproject a sequence of image patterns onto the scene at a pattern framerate; a camera configured and arranged to capture a sequence ofunsynchronized image patterns of the scene at an image capture rate; anda processor configured and arranged to synthesize a sequence ofsynchronized image frames from the unsynchronized image patterns of thescene, each of the synchronized image frames corresponding to one imagepattern of the sequence of image patterns.
 2. The system of claim 1,wherein the sequence of image patterns comprises binary patterns.
 3. Thesystem of claim 1, wherein the number of image patterns in the sequenceof image patterns is less than or equal to the number of unsynchronizedimage frames in the sequence of unsynchronized image frames.
 4. Thesystem of claim 1, wherein the camera has a rolling shutter operation.5. The system of claim 1, wherein the camera has a global shutteroperation.
 6. The system of claim 1, wherein the image capture rate ofthe camera is equal to the pattern frame rate of the projector.
 7. Thesystem of claim 1, wherein the image capture rate of the camera isgreater than the pattern frame rate of the projector.
 8. A method ofcapturing the surface geometry of a three-dimensional object in a scene,comprising: projecting a sequence of image pattern into the scene at apattern frame rate; capturing a sequence of unsynchronized imagepatterns of the scene at an image capture rate; and synthesizing asequence of synchronized image frames from the from the unsynchronizedimage patterns of the scene, each of the synchronized image framescorresponding to one image pattern of the sequence of image patterns. 9.The method of claim 8, wherein the step of projecting a sequence ofimage patterns comprises projecting a sequence of binary patterns. 10.The method of claim 8, wherein the number of image patterns projected inthe sequence of image patterns is less than or equal to the number ofunsynchronized image frames in the sequence of unsynchronized imageframes.
 11. The method of claim 8, further comprising selecting apattern frame rate.
 12. The method of claim 8, further comprisingselecting an image capture rate.
 13. The method of claim 8, wherein theimage capture rate is equal to the pattern frame rate.
 14. The method ofclaim 8, wherein the image capture rate is greater than the patternframe rate.
 15. The method of claim 8, further comprising decoding thesequence of synchronized image frames.
 16. The method of claim 15,further comprising applying three-dimensional triangulation to thesequence of synchronized image frames.
 17. The method of claim 16,further comprising applying geometric processing to the sequence ofsynchronized image frames.
 18. A method to synthesize a sequence ofsynchronized image frames synchronized from a sequence of unsynchronizedimage frames; each of the synchronized image frames corresponding to oneof a sequence of image patterns; the synchronized image frames andunsynchronized image frames being image frames of the same width andheight; the image frames comprising a plurality of common pixels; themethod comprising the steps of: a) partitioning the plurality of commonpixels into a pixel partition; the pixel partition comprising aplurality of pixel sets; the pixel sets being disjoint; each of theplurality of common pixels being a member of one of the pixel sets; b)selecting a pixel set; c) building a pixel set measurement matrix; d)estimating a pixel set projection matrix; the pixel set projectionmatrix projecting a measurement space vector onto the column space ofthe pixel set measurement matrix; e) estimating a pixel set systemmatrix; the system matrix being parameterized by a set of system matrixparameters; f) estimating a pixel set synchronized matrix as a functionof the pixel set measurement matrix and the pixel set system matrix; g)repeating steps b) to f) until all the pixels sets have been selected;and h) constructing a sequence of synchronized image frames from thepixel set synchronized matrices.
 19. A method as in claim 18, where thepixel partition comprises a single pixel set, and the single pixel setcontains all the pixels of the image frames.
 20. A method as in claim18, where the number of pixel sets in the pixel partition is equal tothe height of the image frames, and each row of the image frames is apixel set.
 21. A method as in claim 18, where the pixel set measurementmatrix is modeled as the product of the system matrix times the pixelset synchronized matrix, and the step of estimating the pixel setsynchronized matrices reduces to the solution of a linear least-squaresproblem.
 22. A method as in claim 18, where the system matrix isparameterized by an image frame period parameter, an integration timeparameter, and a first pattern delay parameter.